Extending the usefulness of the verbal memory test: The promise of machine learning

The evaluation of verbal memory is a core component of neuropsychological assessment in a wide range of clinical and research settings. Leveraging story recall to assay neurocognitive function could be made more useful if it were possible to administer frequently (i.e., would allow for the collection of more patient data over time) and automatically assess the recalls with machine learning methods. In the present study, we evaluated a novel story recall test with 24 parallel forms that was deployed using smart devices in 94 psychiatric inpatients and 80 nonpatient adults. Machine learning and vector-based natural language processing methods were employed to automate test scoring, and performance using these methods was evaluated in their incremental validity, criterion validity (i.e., convergence with trained human raters), and parallel forms reliability. Our results suggest moderate to high consistency across the parallel forms, high convergence with human raters (r values ~ 0.89), and high incremental validity for discriminating between groups. While much work remains, the present findings are critical for implementing an automated, neuropsychological test deployable using remote technologies across multiple and frequent administrations

Reflections on the nature of measurement in language-based automated assessments of patients' mental state and cognitive function

Modern advances in computational language processing methods have enabled new approaches to the measurement of mental processes. However, the field has primarily focused on model accuracy in predicting performance on a task or a diagnostic category. Instead the field should be more focused on determining which computational analyses align best with the targeted neurocognitive/psychological functions that we want to assess. In this paper we reflect on two decades of experience with the application of language-based assessment to patients' mental state and cognitive function by addressing the questions of what we are measuring, how it should be measured and why we are measuring the phenomena. We address the questions by advocating for a principled framework for aligning computational models to the constructs being assessed and the tasks being used, as well as defining how those constructs relate to patient clinical states. We further examine the assumptions that go into the computational models and the effects that model design decisions may have on the accuracy, bias and generalizability of models for assessing clinical states. Finally, we describe how this principled approach can further the goal of transitioning language-based computational assessments to part of clinical practice while gaining the trust of critical stakeholders

Towards a temporospatial framework for measurements of disorganization in speech using semantic vectors

Incoherent speech in schizophrenia has long been described as the mind making “leaps” of large distances between thoughts and ideas. Such a view seems intuitive, and for almost two decades, attempts to operationalize these conceptual “leaps” in spoken word meanings have used language-based embedding spaces. An embedding space represents meaning of words as numerical vectors where a greater proximity between word vectors represents more shared meaning. However, there are limitations with word vector-based operationalizations of coherence which can limit their appeal and utility in clinical practice. First, the use of esoteric word embeddings can be conceptually hard to grasp, and this is complicated by several different operationalizations of incoherent speech. This problem can be overcome by a better visualization of methods. Second, temporal information from the act of speaking has been largely neglected since models have been built using written text, yet speech is spoken in real time. This issue can be resolved by leveraging time stamped transcripts of speech. Third, contextual information - namely the situation of where something is spoken - has often only been inferred and never explicitly modeled. Addressing this situational issue opens up new possibilities for models with increased temporal resolution and contextual relevance. In this paper, direct visualizations of semantic distances are used to enable the inspection of examples of incoherent speech. Some common operationalizations of incoherence are illustrated, and suggestions are made for how temporal and spatial contextual information can be integrated in future implementations of measures of incoherence

Reflections on the nature of measurement in language-based automated assessments of patients' mental state and cognitive function

Modern advances in computational language processing methods have enabled new approaches to the measurement of mental processes. However, the field has primarily focused on model accuracy in predicting performance on a task or a diagnostic category. Instead the field should be more focused on determining which computational analyses align best with the targeted neurocognitive/psychological functions that we want to assess. In this paper we reflect on two decades of experience with the application of language-based assessment to patients' mental state and cognitive function by addressing the questions of what we are measuring, how it should be measured and why we are measuring the phenomena. We address the questions by advocating for a principled framework for aligning computational models to the constructs being assessed and the tasks being used, as well as defining how those constructs relate to patient clinical states. We further examine the assumptions that go into the computational models and the effects that model design decisions may have on the accuracy, bias and generalizability of models for assessing clinical states. Finally, we describe how this principled approach can further the goal of transitioning language-based computational assessments to part of clinical practice while gaining the trust of critical stakeholders

Predicting self-injurious thoughts in daily life using ambulatory assessment of state cognition

Self-injurious thoughts (SITs) fluctuate considerably from moment to moment. As such, “static” and temporally stable predictors (e.g., demographic variables, prior history) are suboptimal in predicting imminent SITs. This concern is particularly true for “online” cognitive abilities, which are important for understanding SITs, but are typically measured using tests selected for temporal stability. Advances in ambulatory assessments (i.e., real-time assessment in a naturalistic environment) allow for measuring cognition with improved temporal resolution. The present study measured relationships between “state” cognitive performance, measured using an ambulatory-based Trail Making Test, and SITs. Self-reported state hope and social connectedness was also measured. Data were collected using a specially designed mobile application (administered 4x/week up to 28 days) in substance use inpatients (N = 99). Consistent with prior literature, state hope and social connectedness was significantly associated with state SITs. Importantly, poorer state cognitive performance also significantly predicted state SITs, independent of hallmark static and state self-report risk variables. These findings highlight the potential importance of “online” cognition to predict SITs. Ambulatory recording reflects an efficient, sensitive, and ecological valid methodology for evaluating subjective and objectives predictors of imminent SITs

Modeling remotely collected speech data: Applications for psychiatry

Detecting signs of disorder from listening to spoken words is a core method in psychiatry. Traditionally the interpretation of speech depends on inherently subjective processes. By contrast, digital technology can be leveraged to detect and analyze what words are spoken, timestamp when they are uttered and quantify the manner in which they are expressed. With the use of mobile communication technology, digital speech processing tools are possible to use outside of traditional laboratory settings. This thesis argues that the necessary infrastructure to move speech processing into clinical practice is currently available. To examine this claim, a mobile application for remote mental state assessments was developed that implemented speech-based neuropsychological testing in 353 participants in two countries. It was possible to collect speech data in ecologically valid settings, but future larger scale implementations must solve technical, legal and cultural challenges by interdisciplinary teamwork. The findings of spoken responses on the classic Stroop color-word test from 57 patients with substance use disorders and 86 healthy participants showed that the production of single-word speech utterances could be measured with a high level of temporal precision. The classic Stroop task response latency interference was replicated and the scope of measurements was extended with novel speech characteristics. The audio files from 59 participants naming words in a category fluency task could be analyzed for both the temporal dynamics of response-word sequences and the semantic relatedness between words. Finally, the story recall ability in 25 patients with serious mental illness and 79 healthy participants was examined, and automated measurements of their ability to retell a story was computed using both simple word-count procedures and more advanced estimates of distances in a semantic vector space. In conclusion, it is technologically feasible to develop instruments for measuring multiple aspects of how patients with psychiatric disorders speak, and traditional speech-based neuropsychological tests can be employed outside of a laboratory setting provided the digital infrastructure is able to ensure the privacy of the users

Updating verbal fluency analysis for the 21st century: Applications for psychiatry

Evaluating patients’ verbal fluency by counting the number of unique words (e.g., animals) produced in a short-period (e.g., 1–3 min) is one of the most widely employed cognitive tests in psychiatric research. We introduce new methods to analyze fluency output that leverage modern computational language technology. This enables moving beyond simple word counts to charting the temporal dynamics of speech and objectively quantifying the semantic relationship of the utterances. These metrics can greatly expand the current psychiatric research toolkit and can help refine clinical theories regarding the nature of putative language differences in patients

Using Automated Speech Processing for Repeated Measurements in a Clinical Setting of the Behavioral Variability in the Stroop Task

The Stroop interference task is indispensable to current neuropsychological practice. Despite this, it is limited in its potential for repeated administration, its sensitivity and its demands on professionals and their clients. We evaluated a digital Stroop deployed using a smart device. Spoken responses were timed using automated speech recognition. Participants included adult nonpatients (N = 113; k = 5 sessions over 5 days) and patients with psychiatric diagnoses (N = 85; k = 3–4 sessions per week over 4 weeks). Traditional interference (difference in response time between color incongruent words vs. color neutral words; M = 0.121 s) and facilitation (neutral vs. color congruent words; M = 0.085 s) effects were robust and temporally stable over testing sessions (ICCs 0.50–0.86). The performance showed little relation to clinical symptoms for a two-week window for either nonpatients or patients but was related to self-reported concentration at the time of testing for both groups. Performance was also related to treatment outcomes in patients. The duration of response word utterances was longer in patients than in nonpatients. Measures of intra-individual variability showed promise for understanding clinical state and treatment outcome but were less temporally stable than measures based solely on average response time latency. This framework of remote assessment using speech processing technology enables the fine-grained longitudinal charting of cognition and verbal behavior. However, at present, there is a problematic lower limit to the absolute size of the effects that can be examined when using voice in such a brief ‘out-of-the-laboratory condition’ given the temporal resolution of the speech-to-text detection system (in this case, 10 ms). This resolution will limit the parsing of meaningful effect sizes

Using automated syllable counting to detect missing information in speech transcripts from clinical settings

Speech rate and quantity reflect clinical state; thus automated transcription holds potential clinical applications. We describe two datasets where recording quality and speaker characteristics affected transcription accuracy. Transcripts of low-quality recordings omitted significant portions of speech. An automated syllable counter estimated actual speech output and quantified the amount of missing information. The efficacy of this method differed by audio quality: the correlation between missing syllables and word error rate was only significant when quality was low. Automatically counting syllables could be useful to measure and flag transcription omissions in clinical contexts where speaker characteristics and recording quality are problematic

A Dynamic Method, Analysis, and Model of Short-Term Memory for Serial Order with Clinical Applications

This study examined the robustness of a traditional memory task when moved out of controlled traditional settings. A letter recall task was designed to be self-administered via a smart-device which assessed recall by participants’ writing their responses on the device. This enabled collection of both the letter recalled and the timing of this recall such that the temporal dynamics could be examined. Participants were patients with mental illness (n=71) and healthy volunteers (n=103). Temporal dynamics were examined using a new mechanism that measured memory retrieval time precisely. Data were analyzed for accuracy, time and their relationships. The classic memory phenomena and associated effects were replicated. In terms of temporal dynamics, this is the first demonstration of primacy and recency effects in time domain variables, as well as phonological similarity effects as evident by the inverted U-shaped curves in time. The speed of short-term memory processes affects accuracy, error types and timing. The robustness of these memory effects and new approach to temporal dynamics suggest this framework may be suitable for clinical applications, notably for the long-term monitoring of cognition in patients with mental illness